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The invention relates to a method of segmenting a composite image of pixels 
into a number of fields corresponding to lay-out elements of the image, the pixels 
1 0 having a value representing the intensity and/or color of a picture element. 

The invention further relates to a device implementing the method, which 
device comprises an input unit for inputting an image, and a processing unit 

Several methods for segmenting a composite image, such as a document 
including text and figures, to identify fields corresponding to layout elements, are 
15 known in the art, and a common approach is based on processing the background. 
The image is represented by pixels that have a value representing the intensity 
and/or color of a picture element. The value is classified as background (usually 
white) or foreground (usually black, being printed space). The white background 
space that surrounds the printed regions on a page is analyzed. 

20 

A method for page segmentation is known from the article "Image 
Segmentation by Shape-Directed Covers" by H.S. Baird et.al. in "Proceedings 10th 
International Conference on Pattern Recognition, Atlantic City, NY, June 1990, pp. 
B20-825". A set of maximal rectangles is constructed that are rectangles of 

25 background pixels that cannot be enlarged without including a foreground pixel. 
Relds are defined in the Image based on the set by covering the total image with a 
reduced set of maximal rectangles. The remaining 'uncovered* area is considered 
foreground and fields are found by connected component analysis of the uncovered 
foreground areas. A problem of the method is that the fields are defined as areas in 

30 the pixel domain, which does not allow computationally efficient further processing. 

Further a method for page segmentation is known from the article "Flexible 
page segmentation using the background* by A. Antonacopoulos and R.T Ritchings 
in "Proceedings 12th International Conference on Pattern Recognition, Jerusalem, 
Israel, October 9-12, IEEE-CS Press, 1994, vol2, pp. 339-344". The background 

35 white space is covered with tiles, i.e. non-overlapping areas of background pix Is. 



The contour of a foreground field in the image is identified by tracing along the white j 
tiles that encircle it f such that the inner borders of the tiles constitute the border of a j 
field for further analysis. A probl m of the method is that the borders of the fields are 
represented by a complex description which frustrates an efficient further analysis. 
5 ' I 

it is an object of the invention to provide a method and device for segmenting j 
an image which is more efficient, and in particular delivers a simple description of the j 
segmented image that can easily be used in further processing steps. j 

According to a first aspect of the invention the object is achieved by providing I 
10 a method comprising ! 

- constructing a graph having vertices and edges connecting the vertices, on the ' 
, basis of background areas in the image, said graph edges corresponding to field 

separators that together delineate the fields of the image, 

- constructing a list of contiguous shortest cycles that together completely cover at 
1 5 least a part of the image, a shortest cycle being defined as a closed path from a 

vertex back to that same vertex via the edges of the graph, that has the lowest sum 
of weights of edges of all possible closed paths from said vertex back to said vertex, 
and 

~ defining the shortest cycles of the list as the fields of the image. 

20 According to a second aspect of the invention the object is achieved with a f 

device for segmenting an image of pixels into a number of fields corresponding to \ 
lay-out elements of the image, the pixels having a value representing the intensity 
and/or color of a picture element, which device comprises - ' 
" ~ an input unit for inputting an image, 

25 - a processing unit including 

- a graph constructor for constructing a graph Including vertices and edges 
connecting the vertices, on the basis of background areas in the image, said graph 
edges corresponding to field separators that together delineate the fields of the 
image, . 

30 - a pathfinding module for determining within a graph a closed path from a vertex 
back to that same vertex via the edges of the graph, that has the lowest sum of 
weights of edges of all possible closed paths from said vertex back to said vertex, 
further to be called: a shortest cycle, 
a list module for constructing a list of contiguous shortest cycles that together 

35 completely cover at least a part of the image, and 

- a field definer for defining the shortest cycles of the list as the fields of the image. 
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According to a third aspect of the invention the object is achieved with a 
computer program product for performing the method. 

Construction of a graph has the advantage that the edges provide a compact 
5 and efficient representation of the borders of the fields. Further analysis of the fields 
based on the graph is computationally efficient. However, the step from the graph . 
representation of the image to the fields is not self-evident, because a graph does not 
have an inherent odering of edges and vertices that show the fields. 

The step of constructing a list of shortest cycles is a particularly 
1 0 advantageous way of determining the fields in the image, because the shortest 
cycles already have a shape substantially corresponding with the relevant polygon 
fields in the image. Such fields may have complex forms, although normally they are 
bordered by vertical and horizontal lines, at least in a Manhattan type layout, that is 
predominantly used in newspapers. 
1 5 Thus, the effect of constructing a list of shortest cycles is that the fields can 

be easily identified from the list. 

In one embodiment of the invention, a weight is assigned to an edge, such as 
the Euclidean distance between the vertices of that edge. This weight is used in 
determining the shortest cycles. In this embodiment, the shortest cycles are indeed 
20 the "shortest? ones in Euclidean terms. Also other forms of weights may be 
contemplated, e.g. stepped functions of the distance. 

In an embodiment of the present invention, the step of constructing the list of 
shortest cycles comprises selecting an edge that can be part of at most a single 
shortest cycle, determining the shortest path that connects the vertices of said edge 
25 alternative to said edge, and combining said edge and said shortest path. In this way, 
shortest cycles are efficiently found. 

In a further embodiment, the step of constructing the list of shortest cycles is 
an iterative process, wherein, after finding a shortest cycle, the graph is reduced by 
removing any edge that is part of that shortest cycle and that cannot be part of a 
30 further shortest cycle, and then a next shortest cycle is determined. The advantage is 
that the graph is steadily reduced during the process of constructing the list of 
shortest cycles, which reduces the computational effort. Such an edge that can be 
part of at most a single shortest cycle is for instance an edge at the outer border of 
the graph. 

35 In practice, a field of the image may be completely comprised in a larger field. 

As a consequence, also the shortest cycles corresponding to these fields have this 
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property. Yet, these fields need to be separately processed in the further 
interpretation steps following segmentation- Therefore, in a further embodiment, the 
step of defining the fields comprises checking if a first shortest cycle that encloses a 
first area completely includes a second shortest cycle that encloses a second area 
5 smaller than the first area, and, if so, subtracting the second enclosed area from the 
first enclosed area. 

Alternatively, the list of shortest cycles is sorted on the size of the enclosed 
areas of the shortest cycles and any further processing of the image is performed 
successively on fields corresponding to the shortest cycles in the order of the sorted 
1 0 list Thus, if the smallest fields are processed first, they are automatically excluded 
from the processing of the larger field they are contained in. Such further processing 
Is, e.g. determining the reading order. 

Further preferred embodiments of the device according to the invention are 
given in the further claims. 
1 5 These and other aspects of the invention will be apparent from and elucidated 

further with reference to the embodiments described by way of example in the 
following description and with reference to the accompanying drawings, in which 
Figure 1 shows an overview of an exemplary segmentation method, 
Figure 2 shows a part of a sample Japanese newspaper, 
20 Figure 3 shows the merging of objects along a single direction, 

Figure 4 shows segmentation and two directional merging of objects, * ' 
Figure 5 shows construction of a maximal rectangle from white runs, 
Figure 6 shows construction of maximal white rectangles, 
- Figure 7 shows cleaning of overlapping maximal white rectangles, 
25 Figure 8 shows a graph on a newspaper page, 

Figure 9 shows two types of intersection of maximal rectangles, and 
Figure 10 shows a device for segmenting a picture. 
Figure 1 1 shows a flow diagram for defining fields based on generating a list 
of shortest cycles, 

30 Figures 12a and 12b show graphs and cycles in the graphs, 

Figures 13a and 13b show shortest cycle detection, and 
Figure 14 shows the graph reduction method. 
The Figures are diagrammatic and not drawn to scale. In the Figures, elements 
which correspond to elements already described have the same reference numerals. 
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Figure 1 shows an overview of an exemplary segmentation method, showing 
three basic steps from known segmentation systems. The input image 1 1 is 
processed in a CCA module 14 that analyses the pixels of th image using 
Connected Component Analysis. First an original picture that may be a black-and- 
5 white, grayscale or coloured document, e.g. a newspaper page, is scanned, 

preferably in gray scale. Grayscale scanned pictures are halftoned for assigning a 
foreground value (e,g. black) or a background value (e.g. white) to each pixel- The 
CCA module 1 4 finds foreground elements in the image by detecting connected 
components (CC) of adjacent pixels having similar properties. An example of the first 

1 0 steps in the segmentation process are for instance described in US 5,856,877. The 
CCA module produces as output CC Objects 12, that are connected components of 
connected foreground pixels. An LA module 15 receives the CC Objects 12 as input 
and produces Layout Objects 13 by merging and grouping the CC Objects to form 
larger layout objects such as text lines and text blocks. During this phase, heuristics 

1 5 are used to group layout elements to form larger layout elements. This Is a logical 
step in a regular bottom-up procedure. An AF module 16 receives the Layout Objects 
1 3 as input and produces Articles 17 as output by article formation. In this module, 
several layout objects that constitute a larger entity are grouped together. The larger 
entity is assembled using layout rules that apply to the original picture. For example 

20 in a newspaper page the AF module groups the text blocks and graphical elements 
like pictures to form the separate articles, according to the layout rules of that specific 
newspaper style. Knowledge of the layout type of the image, e.g. Western type 
magazine, Scientific text or Japanese article layouts, can be used for a rule-based 
approach of article formation resulting In an improved grouping of text blocks. 

25 According to the invention additional steps are added to the segmentation as 

described below. The steps relate to segmentation of the image into fields before 
detecting elements within a field, i.e. before forming layout objects that are 
constituted by smaller, separated but interrelated items. Figure 2 shows a sample 
Japanese newspaper. Such newspapers have a specific layout that includes text 

30 lines in both horizontal reading direction 22 and vertical reading direction 21 . The 
problem for a traditional bottom-up grouping process of detected connected 
components is that it is not known in which direction the grouping should proceed. 
Hence the segmentation is augmented by an additional step of processing the 
background for detecting the fields in th page. Subsequently the reading direction 

35 for each field of the Japanese paper is detected before the grouping of characters is 
performed. 



In an embodiment of the method, separator elements, e.g. black lines 23 for 
separating columns are detected and converted into background elements. With this 
option, it is possible to separate large elements of black lines 23 containing vertical 
and horizontal lines that are actually connected into different separator elements, (n 
Japanese newspapers, lines are very important objects for separating fields in the 
layout. It is required that these objects are recognized as lines along separation 
directions. Without this option, these objects would be classified as graphics. Using 
the option the lines can be treated as separator elements in the different orientations 
separately for each separation direction. 

Figure 3 shows a basic method of merging objects in a single direction. The 
Figure depicts the basic function of the LA module 15 for finding the layout objects 
oriented in a known direction, such as text blocks for the situation that the reading 
order is known. Connected components 12 are processed in a first, analysis step 31 
by statistical analysis resulting in computed thresholds 32. In a second, classification 
step 33 the CC-classification is corrected resulting in the corrected connected 
components 34, which are processed in a third, merging step 35 to join characters to 
text lines, resulting in text lines and other objects 36. tn a fourth, text merging step 37 
the text lines are joined to text blocks 38 (and possibly other graphicaJ objects). 
According to the requirements for Japanese news papers the traditional joining of 
objects must be along at least two reading directions, and the basic method 
described above must b© improved therefor. 

Figure 4 shows segmentation and two directional joining of objects. New 
additional steps have been added compared to the single directional processing in 
Figure 3. In a first (pre-) processing step a graph 41 of the image is constructed. The 
construction of the graph by finding field separators is described below. In the graph, 
fields are detected in field detection step 42 by finding areas that are enclosed by 
edges of the graph. The relevant areas are classified as fields containing text blocks 
47. In the text block 47 (using the connected components 43 or corrected connected 
components 34 that are in the text block area) the reading order 45 is determined in 
step 44. The reading direction detection is based upon the document spectrum, e.g. 
on the method of O'Gorman and Kasturi described in "Document Image Analysis" 
IEEE Computer Society Pressr Los Alamitos,- 1995: Using the' fields of the text blocks 
47, the contained connected components 43 and the reading order 45 as input, the 
Line Build step 46 joins the characters to lines as required along the direction found. 

Now the constructing of the graph 41 is described. A graph-representation of 
a document is created using the background of a scan. Pixels in the scan are 



22-11-2002 11 = 17 QCE CORPORATE PftTENTS +31773595497 p ^ 

7 008 22 -iL2002 lf :16: 



classified as background (usually white) or foreground (usually black). Because only 
large areas of white provide information on fields, small noise objects are removed, 
e.g. by down-sampling the image. The down-sampled image may further be de- 
speckled to remove single foreground (black) pixels. 
5 The next task is to extract the important white areas. The first step is to detect 

so-called white runs, one pixel high areas of adjacent background pixels. White runs 
that are shorter than a predetermined minimal length are excluded from the 
processing. 

Figure 5 shows, as an example, four horizontal runs 51 of white pixels, that 
1 0 are adjacent in vertical direction. Foreground area 53 is assumed to have foreground 
pixels directly surrounding the white runs 51 . A "maximal white rectangle* is defined 
as the largest rectangular area that can be constructed from the adjacent white runs 
51 , hence a rectangular white area that can not be extended without including black 
(foreground) pixels. A maximal white rectangle 52 is shown based on the four white 
1 5 runs 51 having a length as indicated by the vertical dotted lines and a width of 4 ' >** 
pixels. When a white rectangle can not be extended it has a so-called maximal 
separating power. Such a rectangle is not a smaller part of a more significant white *!.'. 
area. Hence the rectangle 52 is the only possible maximal rectangle of width 4. 
Further rectangles can be constructed of width 3 or 2. A further example is shown in 
20 Figured 

The construction of white rectangles is done separately in different separation 
directions, e.g. horizontal and vertical white rectangles. Vertical white rectangies are 
detected by rotating the image, and detecting horizontal white runs for the rotated 
image. It is noted that depending on the type of image or application also other 

25 separation directions may be selected such as diagonal. 

An algorithm for constructing maximal white rectangles is as follows. The 
input of the algorithm consists of all horizontal one pixel high white runs (WR) 
detected from a given image. Each white run is represented as a rectangle 
characterized by a set of coordinates ((x u *Ux*yd) t where and Xi and ^ are 

30 coordinates of its top left corner and x^ are the coordinates of its bottom right 

corner. Each white run present in the active ordered object INPUT LIST is tested on 
an extension possibility. The extension possibility Is formulated in the condition 
whether a given WR, labeled by p, can produce a maximal white rectangle (MWR) or 
not If the extension possibility is FALSE, p is already a maximal one, p is deleted 

35 from the active INPUT LIST and written to the active RESULT LIST. If the extension 
possibility Is TRUE, the test for extension is repeated until all MWRs initiated by p 



have been constructed. Then p is deleted from the INPUT LIST and all MWRs 
obtained from p are written to the active RESULT LIST. When all white rectangle 
from the INPUT LIST have been processed, the RESULT LIST will contain all MWKs. 
To increase the efficiency of the algorithm, a sort on the y value is applied to the 
INPUT LIST. First, the algorithm is applied for hori2ontal WRs, i.e. for white runs with 
width larger than height. And after a 90° turn of the image it can be applied to vertical 
WRs. 

In an embodiment the algorithm for constructing the maximal rectangles is as 
follows. The rectangle data are stored as a linked list, with, at least, the coordinates = 
of the rectangle vertices contained in it. The INPUT and RESULT LISTs are stored 
as a linked list too, with, at least, three elements, such as the number of white 
rectangles, and pointers on the first and the last element in the linked list. The 
following steps are executed: Activate INPUT LIST; Initiate RESULT LIST; Initiate 
BUFFER for temporary coordinates of the selected rectangle. Start from the first 
white rectangle, labeled by pi, out of the active ordered INPUT LIST. The next white 
rectangle on the list is labeled by p 2 . For each white rectangle on the INPUT LIST 
examine if p, has extension possibility. For the active white rectangle p 1( find the first 

one labeled by p ni , j=-| I, on the active ordered INPUT LIST, which satisfies 

ya(Pi)=yi(Pni) 

X^Pnj) < Xz(pi) 

Xa(Pnj) 2 x^p,) 

This search results in the set {pm,^,...^. Only if the set {pm ,p n a,...,p n i} is not 
empty, pi is said to have extension possibility. 

- If p, does not have an extension possibility, then p t is a maximal white rectangle. 
Write pi to the RESULT LIST, and remove pi from the INPUT LIST f and proceed 
with pa. 

- If pi is extendible, then apply the extension procedure to ft. Proceed with p a . We 
note here, that can have an extension possibility while being maximal itself. 

The Extension Procedure is as follows. Suppose & has an extension possibility, then 
there is the set {pm.Pn2,...,pni}. The extension procedure is applied to each element of 

{Pni'Pns Pn* consistently. For the white rectangle p, which is extendible with 

rectangle p njl j « 1,...,l, construct a new rectangle p^with coordinates: 
Xi(Pi,nj) - max { xi(pi), x^pnj }, 
x 2 (Pi,nj) * min { x*(pt), x 2 (p n| ) } ■ 

yi(pi.rv) = yi(Pt), 

Y2(Pl.nj) = y 2 (Pnj) 
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Write the coordinates of p 1|lHl j=1 to the "coordinates" buffer. Repeat the test on 
extension possibility now for p^. If the test Is TRUE, p^ is maximal. Write p 1>nj to 
the RESULT LIST, otherwise, extend p w 

Before applying the extension procedure to p^, we check p t and p nJ for 
5 absorption effect The test of p t and p ni for absorption effect with p i.^ is as follows. By 
absorption effect we mean the situation, in which p 1 ( pm) or both is (are) completely 
contained in pi, n j. In coordinates this means: 
XtftHrt) * ^(pj, 

Mp^d £ X2(p k ), where k^i,n if j=1 ,...,!. 
10 If the condition is TRUE for p 1( then pi is absorbed by Remove p^ from the 

INPUT LIST. If the condition is TRUE for p ni , then p nj is absorbed by p iini . Remove p^ 
from the INPUT LIST. 

The algorithm assumes that the rectangle is wider than it is high, and thus the 
rectangles are primarily horizontal. To construct MWRs in vertical direction, the 
1 5 original binary image is rotated by 90° clockwise. The algorithm mentioned above is 
repeated for the rotated image. As a result, all vertical MWRs for the original image 
are constructed. 

Figure 6 shows construction of maximal white rectangles. The pixel 
coordinates are displayed along a horizontal x axis and a vertical y axis. Four white , 

20 runs 61 are shown left in the Figure. The white runs (WR) are described as 

rectangles with the coordinates of their upper and bottom comers correspondingly: 
WR,: ((10,1), (50,2)), 
WR 2 :((10,2),(50,3)), 
WRa : «5,3),(30 f 4)), 

25 WR4 : ((40,3),(60,4)>. 

All maximal white rectangles from these white runs are constructed. The resulting 
five maximal white rectangles (MWR) are shown in the right part of the Figure as 
indicated by 62, 63, 64, 65 and 66. The five MWR shown are the complete set of 
MWR for the WR given In the left part of the Figure. A construction algorithm is as 

30 follows. 

Let the INPUT LIST contain the four white runs 61 . The first element from the 
INPUT LIST is WRi<(10 T 1),(50,2)). Label WR, as p,. Examine p t on the extension 
possibiiity as described above. The first candidate for extension is 
WR a ((10,2),(50,3)). Label WR a as p„i. Extend pi with p ni according to the formula for 
35 extension above, which gives a new rectangle p ltn1 with the coordinates 

((10,1),(50,3)). Test pi and p n i on the absorption e/fecf with p iin i. As follows from 
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absorption test both and p nl are absorbed by pi, n i. Therefore, delete pi and p n i 
from the INPUT LIST. Proceed with pi,m. Test pi.m on the extension possibility, which 
gives the first candidate WR3 ((5,3),(30,4)). Label WR 3 as p t1 . Extend p 1itl1 with p t1 
according to the extension formula. As a result, we obtain a new rectangle p (1nl ) it1 
with the coordinates ((10,1),(30,4)). Test p 1(f ,i with p ti on the absorption effect with 
Pn.BH.ti- The test fails. 

Repeat the test on extension possibility lor Po,ni),ti The test fails, Le. p(i. n i>,ti has no 
extension possibility. It means that pn,ni),ti is maximal. Write P(i, n i),n with the 
coordinates ((10,1), (30, 4)) to the RESULT LIST. 

Proceed again with p 1(M and test it on extension possibility, The second candidate 
WR4 ((40 f 3),(60,4)) is found. Label WFl> as p c . Extend p 1tn1 with p e according to the 
extension formula. As a result, we obtain a new rectangle p ( i. n i) lt 2 with the 
coordinates ((40,1),{50,4)). 

Test pi,ni with Pe on the absorption effect with p^),* . The test fails, i.e. no 
absorption. Repeat test on extension possibility for p<i, n i),e and the test fails, i.e. 
p (1 . n1)i »2 has no extension possibility. It means that p (1( „ 1)iia is maximal. Write 
with the coordinates ((40,1), (50,4)) to the RESULT LIST. 

Test p 1ffl1 again on extension possibility. The test fails and p 1jn1 is maximal. Write p 1inl 
with the coordinates ((1 0,1), (50,3)) to the RESULT LIST. 

Return to the INPUT LIST. The INPUT LIST on this stage contains two write runs, i.e. . 
WFb : ((5,3). (30,4)), WR 4 : ((40,3),(60,4)). Start from WRa, and label it as p 2 . Repeat 
test on extension possibility lor p 2 . The test fails, p 2 is maximal. Write pa with the 
coordinates ((5,3),(30,4)) to the RESULT LIST. Remove p 2 from the INPUT LIST. 
Proceed with WR, and label it as p 3 . Test on extension possibility for p 3 gives us that 
p 3 Is maximal. Write p 3 with the coordinates ((40,3) I {60,4)) to the RESULT LIST. 
Remove p 3 from the INPUT LIST. Finally, the RESULT LIST contains five maximal ■ 
white rectangles, i.e, MWR t ; ((10 y 1),(50,3)) indicated in Figure 6 as 64, MWR 2 : 
«10,1),(30,4)) indicated as 62, MWR3 : ((40,1) ( (50,4)) indicated as 63, and 
MWR4 : ((5,3),(30,4)) as 65, MWR 5 : ((40,3),(60,4)) as 66. 

Figure 7 shows a next step in the method according to the invention, namely 
a cleaning step of overlapping maximal white rectangles. In the cleaning step, plural 
overlapping maxima! white rectangles are consolidated into a single so-called' - 
"Informative Maxima! Rectangle" (IWR) that combines the most relevant properties of 
the original maximal white rectangles, as discussed below In detail. 

The cleaning may further include steps like checking on siz& and spatial 
relation. The upper part of Figure 7 shows, as an example, two maximal white 
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rectangles MWR1 and MWR2. The pair is consolidated into a single Informative. 
White Rectangle IWR in the cleaning step as shown in the lower part of the Figure. 
Th process of detecting overlap and consolidating is repeated until no relevant pairs 
can be formed anymore. A criterion for forming pairs may be the size of the overlap 
5 area. 

Further cleaning steps may include removing thin or short rectangles or 
rectangles that have an aspect ratio below a certain predefined value. The criteria for 
removing are based on the type of image, e.g. a width below a predefined number of 
pixels indicates a separator of text lines and is not relevant for separating fields, and 
1 0 a length below a certain value is not relevant in view of the expected sizes of the 
fields. 

An algorithm for the cleaning step is as follows. The start of the cleaning 
procedure is the whole set of MWRs constructed as described above with reference 
to Figures 5 and 6. The cleaning procedure is applied to discard non-informative 

1 5 MWRs. For this reason a measure of non-informativeness is defined. For example a ] 
long MWR is more informative than a short one. A low aspect ratio indicates a more 
or less square rectangle that Is less informative. Further, extremely thin rectangles, 
which for instance separate two text lines, must be excluded. First, all MWRs are { 
classified as being horizontal, vertical or square by computing the ration between 

20 their heights and widths. Square MWRs are deleted because of their non- : > 
informativeness. For the remaining horizontal and vertical MWRs the cleaning 
technique is applied which consists of three steps: 

- Each MWR with a length or width below a given value is deleted. 

- Each MWR with aspect ratio (AR), defined as the ratio of the longer side length 

25 divided by the shorter side length, below a given value is deleted. 

- For each pair of overlapping horizontal (or vertical) MWRi ((Xi,yi),ta,yg) and 
horizontal (or vertical) MWR 2 ((ai,b,),(aa,ba)), an informative white rectangle IWR is 
constructed with the following coordinates: 

(a) Horizontal overlap: 
30 x^min {x 1t a,}» 

y^maxiy^bO, 
xa=max{x a .a 2 }, 
ya= min { y 2l b 2 }. 

(b) Vertical overlap:} 
35 ^rmaxlx,,^}, 

/i^minly,, b,}, 
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x , a=min{x 2l a z } l 
y*2= max { y 2 , b 2 }. 

This process is repeated for all pairs of overlapping MWRs. The set of MWRs now 
comprises Informative White Rectangles IWRs. These IWRs form the starting point 
5 for an algorithm for segmentation of the image into fields corresponding to the lay-out 
elements. The IWRs are potential field separators and are therefore called 
"separating elements". Using the IWRs, the algorithm constructs a graph for further 
processing into a geographical description of the image. 

Figure 8 shows such a graph on a newspaper page. The picture shows a 

10 down-sampled digital image SO of a newspaper page. The original text is visible in 
black in a down-sampled version corresponding to Figure 2. The informative 
rectangles IWR constituting separating elements are shown in gray. For the 
construction of the graph, intersections of separating elements constituted by 
horizontal and vertical white IWRs are determined. The intersection point of two 

1 5 IWRs is indicated by a small black square representing a vertex or vertex 81 in the 
graph. Edges 82 that represent tines that separate the fields in the page are 
constructed by connecting pairs of vertices 81 via "field separators". The edges 82 of 
the graph are shown in white. The distance between the two vertices of an edge, i.e. 
the length, is assigned as weight to the edge for further processing. In an alternative 

20 embodiment a different parameter is used for assigning the weight, e.g. the colour of 
the pixels. An algorithm for constructing the graph is as follows. 

At the beginning, the following notation and definitions for IWRs is given. Let 
R = {r u ».jCni be the non-empty and finite set of all IWRs obtained from a given image 
. - I, where each IWR is specified by its x- and y- coordinates of top left corner and 

25 bottom right comer ( yi w ), (x 2 w , y 2 w ) ), t = 1 ,2,. .., m respectively. Each 
rectangle r T is classified as horizontal, vertical or square based on the ratio of its 
height and width. H = { rv..,h|), V = { vi,... v Vk} , and S = {s rf ...,s d } denote the 
subsets of horizontal, vertical and square IWRs, respectively, such that 
H uVuS = R and m = I + k + d, and 

30 HnV-0, VnS=0, HnS = 0 

where it is assumed that 
H *0, V*0. 

Further the contents of S are ignored and only the subsets H and V are used. This is 
based on the consideration, that in most cases white spaces that form the border of 
35 text or non-text blocks are oblong vertical or horizontal areas. Let h be part of H with 
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coordinates ((x 1s yi),(x 2 ,y 2 )) and v in V with coordinates ((aubM^bz)). Then h and v 
have overlap if 

{x,sa 2 
yi*b z 
y 2 *o 1 

By the intersection point of h and v in case of overlap, we take the unique point P 
10 defined by the coordinates: 

{x P = 1& (maxix^aO + minix^aa}), 
y P = Vk ( max { , b n } + min {y 2 , b 2 } ) 

1 5 For IWRs only two from all possible types of overlap occur, namely overlap resulting . 
In a rectangle and overlap resulting in a point. Line overlap cannot occur, because 
this would be in contradiction with the concept of the MWRs. 

Figure 9 shows two types of intersection of maximal rectangles. For 
constructing the graph the intersection points of vertical and horizontal informative 

20 maximal rectangles are determined to find the position of vertices of the graph, i.e. to 
determine the exact coordinates of the vertices. The left part of the Figure shows a 
first type of intersection of vertical IWR v and a horizontal IWR h, which results in a 
rectangular area 88 with a center of intersection point P. The right part of the Figure 
shows a second type of intersection of a vertical IWR v and a horizontal IWR h, that 

25 results in a single intersection point 89 with a center of intersection at P\ 

An algorithm for constructing the graph based on the intersection points is as 

follows. 

P ■ {Pi,-,Pn} denotes the set of all intersection points of vertical IWRs and 
horizontal IWRs where each p in P is specified by its x~ and y- coordinates (Xp, y p ), 

30 where p=i N, Let the set P be found, and G^X, A) an undirected graph having 

correspondence to P. The graph G=(X,A) consists of a finite number of vertices X 
which are directly related to the intersection points and a finite number of edges A 
which describe the relation between intersection points. Mathematically this Is 
expressed as 

35 

Q(P) = (X(P),A(PxP)), 



P:HxV-Mx P ,y P }, 

where 

XS{1 N}and 

A = ({1 N}x{1,....,N})wlth 

5 

A ( i, j ) = f «», H i and j are not 4-chain connected, 
1 d i|, If i and j are 4-chain connected 

where d p indicates the Euclidean distance between points I and j, and where 4-chain 

1 0 connected means that the vertices of a rectangular block are connected in four 
possible directions of movement. In the above two points I and j are 4-chain 
connected if they can be reached by walking around with the aid of 4-connected 
chain codes with min dij in one direction. 

The graph as constructed may now be further processed for classifying the 

1 5 areas within the graph as text blocks or a similar classification depending on the type 
of picture. In an embodiment the graph is augmented by including foreground 
separators, e.g. black lines or patterned lines such as dashed/dotted lines, in the 
analysis. Also, edges of photos or graphic objects which are detected can be 
included in the analysis. 

20 The present segmenting method may also include a step of removing foreground 
separators. First, foreground separators are recognized and reconstructed as single 
objects. The components that constitute a patterned line are connected by analyzing 
element heuristics, spatial relation heuristics and line heuristics, i.e. building a 
- combined element in a direction and detecting if it classifies as a line. A further 

25 method for reconstructing a solid line from a patterned line is down-sampling and/or 
using the Run Length Smoothing Algorithm (RLSA) as described by K.Y. Wong, R.G. 
Casey, F.NL Wahl in "Document analysis system* IBM J. Res. Dev 26 (19B2) 647- 
656. After detecting the foreground separators they are replaced by background 
pixels. The effect is that larger maximal white rectangles can be constructed, or 

30 supporting any other suitable method using the background pixel property for finding 
background separators. 

Fgufe 11 shows~af^ 
defining fields in an image of pixels, based on generating a list of shortest cycles in a 
graph representation of the image. Basically, the method first constructs a 

35 mathematical description of the image of pixels in the form of a graph having vertices 
and edges connecting the vertices. Then, the graph is broken up into contiguous 
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areas having a smallest possible perimeter, I.e. areas that are delineated by smallest 
possible cycles formed by edges of the graph. Finally, fields are defined in th pixel 
domain of the original image, in correspondance with the (mathematically defined) 
smallest cycles. This approach of searching for smallest possible cycles In the graph 
5 has the advantage, that the corresponding calculations are simple and fast Yet, the 
results are not always entirely correct, when a first shortest cycles completely 
contains a second, smaller shortest cycle. See, for example, Fig. 12b, where a 
shortest cycle 104a completely contains two smaller shortest cycles (around the 
rectangles 103a and 103b). Below we will show that this effect does not lead to 
10 problems. 

In a GPAPH step S1 a graph is constructed having edges that correspond to 
background areas separating the fields in the image. The vertices in the graph 
correspond to angular points of the fields in the image. The fields in the Image are for 
example text blocks in a newspaper page. Basically, the fields are polygon areas 
1 5 having mainly vertical and horizontal borders, but they are not necessarily • *£ 

rectangular. For instance, L-shaped text blocks appear frequently in practical i 
situations. 

-j . ■ 

An example of constructing a graph has been described above with reference 
to Figures 8 and 9. Any other way of constituting a graph for representing an image 

20 that ha9 fields is suitable, provided that the graph has edges corresponding to : the 

actual borders of the fields in the original image. It is noted that the task of finding a »■' 
field in the graph consists of identifying the edges and/or vertices that are bordering . ; ; 
that field. This is not evident from the graph as such, because the graph does not 
have an inherent ordering of edges and vertices that show the fields. Thereto the 

25 following steps are performed. 

in a WEIGHT step S2 a weight is assigned to each edge. The weight of an 
edge in a graph of a layout type image to be segmented may be the Euclidian 
distance between the vertices, i.e. its length in the original image. In an embodiment 
for a different type of application where the image represents a different structure, 

30 e.g. a map, a suitable measure for weight may be traveling time, or a different 
parameter like cost in the weight assigning step the weight for each edge Is 
calculated and added to the graph data-structure. 

In the following steps shortest cycles are determined. A shortest cycle is 
defined as a closed path in the graph via edges that are connected by vertices, which 

35 path has the lowest sum of weights accumulated by adding the weights of the edges 
involved. In an INIT step S3 a list of shortest cycles is created having an initially 
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empty content, and a first starting point, a vertex or edge, within the graph is 
determined. For example for this purpose the graph is sorted and the first starting 
point is set to be the upper left corner vertex of the graph. The process now enters a 
loop of steps S4, S5, S6 and S7, in which a shortest cycle is constructed starting 
5 from the preselected starting point. At the end of the loop the NEXT step S7 selects a 
new starting point for finding a next shortest cycle. The status of the vertices or 
edges in the graph for still being available as candidate starting point is maintained. If 
no candidate starting points are remaining, step S7 terminates the loop and proceeds 
to the next step S8. 

1 0 Within the loop the CYCLE step S4 determines a shortest cycle for the 

selected starting point. The shortest cycle algorithm may construct every possible 
- - cycle and select the shortest cycle from the set The shortest cycle found is judged in 
NEW step S5 for being new with respect to the list of shortest cycles already found. If 
the shortest cycle is new, the ADD step S6 is performed for adding the latest shortest 

1 5 cycle to the list of shortest cycles. Suitable algorithms for constructing the shortest 
cycle, selecting and maintaining starting points and constructing the list of new 
shortest cycles are described below. 

At the time that NEXT step S7 detects that no further candidate starting points 
are available, the list contains the shortest cycles that can be formed in the graph. 

20 The NEXT step S7 may also be arranged for detecting if a required specific part of 
the graph has been fully processed. Such a required part can be defined it it is known 
from the application that only fields contained in a specific part of the graph need to 
be constructed. The list of shortest cycles will then contain the shortest cycles in that 
r part of the graph only. 

25 In a FIELD step S8 the fields are defined on the basis of the list of shortest 

cycles. A field is a polygon represented by the vertices and edges in the shortest 
cycle that encloses the field. The step SB may also include additional steps for 
detecting shortest cycles that contain other, smaller shortest cycles, and for 
correcting the initial results for that case. 

30 The process Is completed at the END step S9. 

In an embodiment the FIELD step S8 includes an additional step for defining 
the fields. First it is determined If a shortest cycle that encioses a larger area further 
contains a smaller area that itself is enclosed by a second shortest cycle. The 
corresponding larger field is corrected by subtracting the smaller enclosed area from 

35 the larger enclosed area. Alternatively the field corresponding to the second shortest 
cycle is processed before processing the field corresponding to the first shortest 
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cycle. When processing the latter the first area is skipped. The additional step is 
further explained below with referenc to Figure 12b. 

Figures 12a and 12b show graphs and cycles in the graphs. The graph is 
constituted by edges 100 and vertices 101 . Each area enclosed by edges 100 
5 corresponds to a field in the original image as represented by the graph. Figure 1 2a 
shows a shortest cycle 102 around a text area 105, Figure 12b shows a further 
graph that has a further text area 104 enclosed by a rectangular shortest cycle 104a. 
The area 1 04 encloses a further* smaller text area 1 03a and a second smaller area 
103b. The smaller areas are also enclosed by shortest cycles. This situation may be 
1 0 handled in one of two alternative approaches. 

In a first approach, the areas are sorted on size of their enclosed area before 
determining further properties of the fields like reading direction. In particular the 
reading order In the smaller text block is determined first. During processing the 
larger shortest cycle the area of the smaller cycle is skipped, such that characters in 
1 5 that area do not influence the determination of reading direction in the larger area. \v u 
In a second approach, the FIELD step S8 comprises detecting such inclusion , 
and compensating for it. In a first sub-step it is determined if a larger shortest cycle 
Includes a smaller shortest cycle. Comparing the enclosed areas and/or the location 
of the shortest cycles reveals the enclosure. For this purpose shared edges or 
20 vertices may be used. In a second sub-step the enclosed area of the smaller shortest 
cycle is subtracted from the enclosed area of the larger shortest cycle. 

Shortest cycles can be detected in several ways, for example using the 
following algorithm. 

Figures 13a and 13b show shortest cycle detection. A method is shown for 
25 finding a shortest cycle based on selecting a starting edge 1 08. In Figure 1 3a a graph 
is shown that is temporarily reduced by removing starting edge 108 between the 
vertices u and v. Now a shortest path in the graph is constructed from vertex u to 
vertex v. Figure 13b shows constructing the shortest cycle by resetting the edge 109. 
Hence the shortest cycle including the edge and vertices u and v is found. Suitable 
30 algorithms for constructing the shortest path between two vertices In a graph can be 
found in "Graph theory: an algorithmic approach" by N. Christofides, Academic 
Press, New York, 1975, in particular Dijkstra's algorithm described therein. By 
subsequently taking edges as starting point a list of shortest cycles is constructed 
covering the total of the graph. In an embodiment the constructing of the list of 
35 shortest cycles comprises only including a shortest cycle on the list if that cycle is not 
already included in the list. 
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The amount of calculations can be drastically limited when a so-called graph 
reduction method is applied. According to this method, a shortest cycle that is 
detected is removed from the graph, and detecting shortest cycles is continued within 
the reduced graph, preferably next to the removed cycle. In this way, the graph is 

5 successively reduced until nothing remains of it. 

A suitable algorithm for constructing the list using graph reduction is shown in 
Figure 14. In a first step 111, each edge is assigned a potential number indicating the 
maxima! number of shortest cycles it can be part of, normally 1 for an edge situated 
at an outer border of the graph and 2 for an edge situated within the graph borders. 

10 In the upper left area a shortest cycle 1 1 5 is shown detected. Once detected, the 

number of potential cycles of the 4 edges included in this shortest cycle is reduced by 
" 1. In the second step 112 the upper left edges that have a potential number of 0, are 
removed from the graph. A second shortest cycle 1 16 is detected in the second area 
to the right, adjacent to the area of cycle 115. In a third step 113 again the potential 

1 5 numbers are reduced and two more sides are removed from the graph. A third 
shortest cycle 1 17 is detected on the second row left. 

Every time a shortest cycle is detected, it is added to a list of shortest cycles 
for the image. The process of detecting a shortest cycle and reducing the graph 
continues until no edges are left. It is noted the dead ends in the graph are removed 

20 also, because they can never be part of a cycle. 

In another embodiment of the method the determining of the shortest cycle is , 
performed by constructing a minimal spanning tree. Such a minimal spanning tree 
represents all shortest paths from a root vertex to the other vertices. The root vertex 
" "u is the first vertex of the removed edge. The tree is used to find the shortest path to 

25 the other vertex v of the removed edge. A data structure of pointers is used to 

represent the minimal spanning tree. The tree is constructed by adding a vertex per 
Iteration, which vertex has the shortest distance to the root vertex of all vertices not 
yet in the tree. The respective new vertex is found by checking the neighbors of all . 
vertices that are already included in the tree and calculating the distance to the root if 

30 that neighbor is not yet in the tree. As soon as the vertex v is added to the tree the 
process is stopped. Advantageously a separate list with the neighbors of the current 
iteration can be maintained 

The minimal spanning tree may also' be represented in variables of the data 
structure that represents the graph, in particular associated with the vertex elements 

35 of the graph. Each vertex element has additional variables to store the tree. For 
finding a new shortest cycle the variables must be initialized, in an embodiment the 
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variables are not individually initialized, but an additional variable is added to each 
vertex that is Initialized before th first tree Is constructed. For each subsequent tree 
a different value is stored in the additional variable, identifying the respective vertex 
as being part of the current tree. Constructing the tree using variables in the graph is 
advantageous in that no separate data structure needs to be maintained for the 
minimal spanning tree. 

Bgure 10 shows a device for segmenting a picture using a method according 
to the invention. The device has an input unit 91 for entering a digital image. The 
input unit may comprise a scanning unit for scanning an image from paper such as 
an electro-optical scanner, or a digital communication unit for receiving the image 
from a network like internet, or a playback unit for retrieving digital Information from a 
record carrier like an optical disc drive. The Input unit 91 is coupled to a processing 
unit 94, which cooperates with a memory unit 92. The processing unit 94 may 
comprise a general purpose computer central processing unit (CPU) and supporting 
circuits and operates using software for performing the segmentation as described; \ 
above. Thereto it includes a GRAPH module for constructing a graph on the basis of ; ; 
background areas in the Image, a WEIGHT module for assigning a weight to the V 
edges of the graph, a PATH module for determining within the graph closed paths or < 
cycles via the edges and determining shortest cycles therefrom, a LIST module for 
constructing a list of contiguous shortest cycles, and a FIELD module for defining the 
shortest cycles of the list as the fields of the image. It Is to be understood that these 
modules may be implemented as program modules in the processing unit 94, 
Therefore, these modules have been drawn with broken lines in Fig. 10. The 
processing unit may also include a user interface 95 provided with control means 
such as a keyboard, a mouse device or operator buttons. The output of the 
processing unit is coupled to a display unit 93. In an embodiment the display unit is a 
printing unit for outputting a processed image on paper, or a recording unit for storing 
the segmented image on a record carrier like a magnetic tape or optical disk. 

Although the invention has been mainly explained by embodiments using a 
Japanese newspaper page as the digital image to be segmented, the invention is 
also suitable for any digital representation of any text or image having a layout in 
fields on a background, such as electrical circuits in layout images for IC design or 
streets and buildings on city maps. Also for a very different application such as 
detecting closed sub-circuits in electric chains this approach may be used. Further it 
is noted that the graph as starting point for executing the segmenting by shortest 
cycles may be constructed differently than the graph described above based on the 
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MWR system. For example a graph may be constructed using tiles as described in 
the article by Antonacopoulos mentioned above, although the particular graph 
disclosed in Antonacopoulos is different from that of the present invention and cannot 
be used. Further the weight assigned to an edge in the graph is not necessarily the 
distance. It must be selected to correspond to a contribution to the shortest cycle, for 
example the weight may be the surface of the tile. It is noted, that in this document 
the use of the verb 'comprise' and its conjugations does not exclude the presence of 
other elements or steps than those listed and the word 'a' or 'an' preceding an 
element does not exclude the presence of a plurality of such elements, that arty 
reference signs do not limit the scope of the claims, that the invention and every unit 
or means mentioned may be implemented by suitable hardware and/or software and 
that several 'means' or 'units' may be represented by the same item. Further, the 
scope of the invention is not limited to the embodiments, and the invention lies in 
each and every novel feature or combination of features described above. 
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CLAIMS 



1 . Method of segmenting an image of pixels into a number of fields 
corresponding to lay-out elements of the image, the pixels having a value 
representing the intensity and/or color of a picture element, 
which method comprises 
1 0 - constructing a graph having vertices and edges connecting the vertices, on the 
basis of background areas in the image, said graph edges corresponding to field 
separators that together delineate the fields of the image, 

- constructing a list of contiguous shortest cycles that together completely cover at 
least a part of the image, a shortest cycle being defined as a closed path from a 

15 vertex back to that same vertex via the edges of the graph, that has the lowest sum 
of weights of edges of all possible closed paths from said vertex back to said vertex, 
and 

- defining the shortest cycles of the list as the fields of the Image. 

20 2. Method as claimed in claim 1 , further comprising 

- assigning a weight to an edge in accordance with a predetermined property thereof, 
such as the Euclidean distance between the vertices of that edge, 

3. Method as claimed in claim 1 , wherein said step of constructing of the fist of 
25 shortest cydes comprises 

- selecting an edge that can be part of at most a single shortest cycle, 

- determining the shortest path that connects the vertices of said edge alternative to 
said edge, and 

-combining said edge and said shortest path. 
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4. Method as claimed in claim 3, wherein an edge that can be part of at most a 
single shortest cycle is an edge at the outer border of the graph. 

5. Method as claimed in any of the claims 1 to 4, wherein said step of 
constructing the list of shortest cycles is an iterative process, wherein, after finding a 
shortest cycle, the graph is reduced by removing any edge that is part of that shortest 



c 



22 



cycle and that cannot be part of a further sh rtest cycle, and then a next shortest 
cycle is determined. 

6. Method according to claim 5, wherein said step of constructing the list of 

5 shortest cycles is terminated when the graph does not have any remaining edges. 

7. Method as claimed in any of the claims 1 to 6, wherein a shortest cycle is 
determined by constructing a minimal spanning tree, which represents all shortest 
paths 1rom a root vertex to the other vertices, and the minimal spanning tree is 

10 represented in variables associated with the vertices of the graph. 

8. Method as claimed in claim 1 , wherein said step of defining the fields 
comprises 

- checking if a first shortest cycle that encloses a first area completely includes a 

1 5 second shortest cycle that encloses a second area smaller than the first area, and 

- subtracting the second enclosed area from the first enclosed area. 

1 o. Method as claimed in claim 1 , wherein said step of defining the fields 
comprises 

20 - determining an enclosed area for each shortest cycle in the list, 

- sorting the list of shortest cycles on the size of the enclosed area, 

and wherein any further processing of the image is performed successively on fields 
corresponding to the shortest cycles in the order of the sorted list. 

25 11. Computer program product for segmenting an image of pixels into a number 
of fields, which program is operative to cause a processor to perform the method as 
claimed in any of the claims 1 to 10. 

12. Device for segmenting an image of pixels into a number of fields 
30 corresponding to lay-out elements of the Image, the pixels having a value 

representing the intensity and/or color of a picture element, which device comprises 

- an input unit (91) for inputting an image, • - - 

- a processing unit (94) including: 

- a graph constructor for constructing a graph including vertices and edges 

35 connecting the vertices, on the basis of background areas in the image, said graph 
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edges corresponding to field separators that together delineate the fields of the 
Image, 

- a pathfinding module for determining within a graph a closed path from a vertex 
back to that same vertex via the edges of the graph, that has the lowest sum of 

5 weights of edges of all possible closed paths from said vertex back to said vertex, 
further to be called: a shortest cycle, 

- a list module for constructing a list of contiguous shortest cycles that together 
completely cover at least a part of the image, and 

- a field definer for defining the shortest cycles of the list as the fields of the image. 

10 

13. Device as claimed in claim 12, wherein the processing unit (94) further 
comprises 

- a weight assigner for assigning a weight to an edge in accordance with a 
predetermined property thereof, such as the Euclidean distance between the vertices 4 

15 of that edge. r 

14. Device as claimed in claim 12, wherein the device comprises a display unit I 
(93) for displaying fields of the image after segmenting. 

20 15. Device as claimed in claim 12 or 13, wherein the list module is arranged for \ . 
iterativety finding shortest cycles by 

- removing any edge from the graph that is part of a shortest cycle to be included on r 
the list and that cannot be part of a further shortest cycle, and 

- terminating constructing the list when the graph does not have any remaining 
25 edges. 
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ABSTRACT 



5 A method is described for segmenting an imagB of pixels into a number of fields. First 
the method finds field separators using the background of the image, in particular 
white areas on a newspaper page. Based on the areas in the image a graph is 
constructed, that has edges corresponding to the white areas and vertices where 
vertical and horizontal white areas intersect. The segmenting starts with assigning 
1 0 weights to the edges, in particular a weight indicating the Euclidean distance 

between the vertices. Then a list of shortest cycles is constructed via the edges and 
vertices in the graph. The fields are defined by the vertices and edges of the shortest 
cycles of the list. 

15 

(Fig. 12a) 
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